Software testing is a process that is used to verify the correctness, completeness, and quality of computer software. It is also used to identify the security features of computer software. In software testing, two broad approaches are used—white-box testing and black-box testing.
In white-box testing, the test developer has knowledge of the internal structure of the source code of the target software to be tested. The test developer has access to the source code of the target software and can insert the code that can link to libraries, which implement the majority of system services. There exist coverage criteria in white-box testing. Examples of coverage criteria include a statement-coverage criterion, a branch-coverage criterion, and a path-coverage criterion. The use of the statement-coverage criterion determines whether each executable statement in the target software code is covered. The branch-coverage criterion is used to identify and validate all the branches in the target software code and ensure that no branching results in the target software behaving in an undesirable manner. The path coverage criterion reports whether each of the possible paths in each function of the target software code has been tested.
In black-box testing, also referred to as specification-based testing or behavioral testing, the test developer does not have access to the source code of the target software to be tested. The test developer is able to access the target software through the same interface that a user is able to access. More specifically, in the case of black-box testing, the test developer knows only about the inputs and the expected outcome corresponding to these inputs. The test developer does not know how the target software arrives at these expected outputs. In black-box testing, coverage criteria such as statement coverage, branch coverage, and path coverage are not widely used.
There are several methods that are widely used in black-box testing. One of these methods involves the equivalence class-partitioning method. In this method, the test developer partitions the input domain of arguments pertaining to the function being tested into equivalence classes. Consequently, if target software is tested for one member of an equivalence class, it is equivalent to it being tested for all the members of the equivalence class. Another method involves boundary value analysis, which involves the creation of test cases around the corners of values in loop conditions, and validating these test cases. For example, in a system or a program that accepts as input a number between one and 10, boundary value analysis would indicate that test cases should be created for the lower and upper bounds of the input domain (1, 10), as well as values just outside these bounds (0, 11), to ensure proper functionality.
Yet another method involves the use of the Category Partition Method (CPM), based on an extended context-free grammar (ECFG) model. The CPM is implemented by means of a Test Specification Language (TSL). A test generator accepts test designs in a TSL format as an input and generates executable test programs, which are also referred to as test frames. A category in the CPM can hold the same value for all the references of the category. A partition can hold different values for different references of the partition. The choices associated with the categories and partitions represent equivalence classes. The method using CPM involves both choice and group coverage. If there are three categories, each consisting of three choices in a program, choice coverage may imply nine test frames, ensuring that all the six choices are used in the emitted test frames. In group coverage, each and every choice of a first category is combined with each and every choice of a second category, thereby producing 27 test frames. The method using CPM involves random selection of choices for associated categories, and partitions and evaluating pre-conditions that are associated with these choices. Typically, a precondition is a Boolean expression that is made up of variables introduced in the test design and/or Boolean expressions specifying the selection status of the choices associated with non-terminals.
The methods mentioned above have one or more of the following limitations. First, if a randomly selected choice does not satisfy associated pre-conditions for a particular category or a partition, back-tracking is required. This process introduces overheads, as choices for already validated categories, or the partitions may need to be validated again. Second, in the above-mentioned methods, especially in CPM after the test generator for the test design in TSL generates the executable test programs, it is not confirmed whether all the test scenarios have been covered. Third, these methods do not provide specific guidelines for choosing inputs and do not provide any framework for a test developer.